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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  & 
Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 

•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters, 
Department  of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test,  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 

James  B.  Greene  Jr.  Keith  F.  Snider,  PhD 

Rear  Admiral,  U.S.  Navy  (Ret.)  Associate  Professor 
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Weapons  Acquisition  Reform:  Reform  Act  Is  Helping  DoD 
Acquisition  Programs  Reduce  Risk,  but  Implementation 

Challenges  Remain 


Michael  J.  Sullivan — Sullivan  is  the  director,  Acquisition  and  Sourcing  Management,  U.S. 
Government  Accountability  Office.  This  group  has  responsibility  for  examining  the  effectiveness  of  the 
DoD’s  acquisition  and  procurement  practices  in  meeting  its  mission  performance  objectives  and 
requirements.  In  addition  to  directing  reviews  of  major  weapon  system  acquisitions  such  as  the  Joint 
Strike  Fighter,  F-22,  Global  Hawk,  and  various  other  major  weapon  acquisition  programs,  Sullivan 
has  developed  and  directs  a  body  of  work  examining  how  the  DoD  can  apply  best  practices  to  the 
nation's  largest  and  most  technically  advanced  weapon  systems  acquisition  system.  This  work  has 
spanned  a  broad  range  of  issues  critical  to  the  successful  delivery  of  systems,  including  technology 
development,  product  development,  transition  to  production,  software  development,  program 
management,  requirement-setting,  cost  estimating,  and  strategic  portfolio  management.  The  findings 
and  recommendations  from  this  work  have  played  a  major  role  in  the  department’s  recent  acquisition 
policy  revisions.  Most  recently,  he  has  directed  the  GAO’s  annual  assessment  of  major  weapon 
systems  programs  for  the  Congress  and  GAO’s  work  with  Congress  in  establishing  acquisition  policy 
reforms.  His  team  also  provides  the  Congress  with  early  warning  on  technical  and  management 
challenges  facing  these  investments.  Sullivan  has  been  with  the  GAO  for  24  years.  He  received  a 
bachelor's  degree  in  political  science  from  Indiana  University  and  a  master’s  degree  in  public 
administration  from  the  School  of  Public  and  Environmental  Affairs,  Indiana  University. 
[sullivanm@gao.gov] 

Introduction 

For  several  decades,  the  GAO  has  reported  on  poor  outcomes  encompassing  cost 
and  schedule  growth  on  the  Department  of  Defense’s  (DoD’s)  major  weapon  acquisition 
programs.  Many  problems  can  be  traced  to  a  culture  where  the  military  services  begin 
programs  with  inflexible  requirements,  immature  technologies,  and  overly  optimistic  cost  and 
schedule  estimates.  Given  pressures  to  reduce  spending  across  the  government,  including 
the  DoD,  finding  ways  to  prevent  or  mitigate  cost  growth  is  crucial  to  U.S.  national  security. 

A  solid  program  foundation  using  good  developmental  testing  and  systems  engineering,  and 
reliable  cost  estimates  is  needed  in  order  to  help  avoid  cost  overruns  and  promote  better 
acquisition  outcomes.  There  have  been  numerous  attempts  in  the  past  to  improve  DoD 
acquisition  outcomes,  including  the  Packard  Commission  (President’s  Blue  Ribbon 
Commission  on  Defense  Management,  1986),  the  Goldwater-Nichols  Act  in  the  1980s 
(1986),  and  the  Federal  Acquisition  Streamlining  Act  of  1994.  More  recently,  Congress 
passed  the  Weapon  Systems  Acquisition  Reform  Act  of  2009  (Reform  Act)1  to  improve  the 
way  weapon  systems  are  acquired  and  avoid  cost  and  schedule  overruns. 

In  2009,  the  Senate  Armed  Services  Committee  asked  us  to  begin  reporting  on  the 
DoD’s  implementation  of  Reform  Act  provisions  and  the  impact  the  Reform  Act  has  had  on 
weapon  acquisition  programs.  This  is  our  third  report  addressing  these  topics.  The  first 
report  focused  on  the  DoD’s  initial  efforts  to  implement  Reform  Act  provisions  for  systems 
engineering  and  developmental  testing,  including  the  placement  of  new  offices  for  these 
activities  within  the  Office  of  the  Secretary  of  Defense  (OSD;  GAO,  2010).  Our  second 
report  examined  the  challenges  the  Services  face  as  they  try  to  strengthen  systems 
engineering  and  developmental  testing  activities  on  weapon  acquisition  programs  (GAO, 


1  As  amended  by  the  Ike  Skelton  National  Defense  Authorization  Act  for  Fiscal  Year  201 1 ,  Pub.  L.  No. 
1 11-383  §§  813  and  1075,  and  the  National  Defense  Authorization  Act  for  Fiscal  Year  2012,  Pub.  L. 
No.  112-81  §§819  and  837. 
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2011b).  This  report  examines  (1)  the  DoD’s  progress  in  implementing  Reform  Act 
provisions;  (2)  the  impact  the  Reform  Act  has  had  on  specific  acquisition  programs;  and  (3) 
challenges  remaining  in  improving  the  weapons  acquisition  process. 

We  conducted  this  performance  audit  from  January  2012  to  December  2012  in 
accordance  with  generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to 
provide  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 
We  believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 

Background 

In  May  2009,  Congress  passed  the  Reform  Act  in  an  effort  to  improve  the  way 
weapon  systems  are  acquired  and  avoid  further  cost  overruns  on  such  programs.  When 
signing  the  Reform  Act  into  law,  the  President  stated  that  its  purpose  was  to  limit  weapon 
system  cost  overruns  and  that  it  would  strengthen  oversight  and  accountability  by  appointing 
officials  who  will  closely  monitor  the  weapons  systems  acquisition  process  to  ensure  that 
costs  are  controlled. 

Four  offices  were  established  as  a  result  of  the  Reform  Act:  SE,  DT&E,  CAPE,  and 
PARCA.  The  SE  and  CAPE  offices  existed  under  other  organizational  titles  prior  to  the 
Reform  Act.  Staffing  levels,  following  the  Reform  Act,  remained  relatively  stable  for  both  of 
these  offices,  but  some  reorganization  was  necessary  to  reflect  new  Reform  Act 
responsibilities.  The  DT&E  and  PARCA  offices  were  newly  established.  The  key  roles  and 
responsibilities  of  these  four  offices  as  outlined  in  the  Reform  Act  are  explained  in  Table  1 . 
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Table  1 .  Key  Responsibilities  of  Offices  Established  as  a  Result  of  the 

Reform  Act 


Office _ Primary  responsibilities _ 

Systems  Engineering  •  serves  as  principal  advisor  to  the  Secretary  of  Defense  and  the  Under 

Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  on  systems 
engineering  activities  in  the  department 

develops  systems  engineering  and  development  planning  guidance  for  the 
DoD 

reviews  and  approves  major  defense  acquisition  program  systems  engineering 
plans 

monitors  major  defense  acquisition  program  systems  engineering  activities 

reports  to  Congress  annually  on  systems  engineering  organization, 
capabilities,  and  activities 

serves  as  principal  advisor  to  the  Secretary  of  Defense  and  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  on 
developmental  test  and  evaluation  activities 
develops  developmental  test  and  evaluation  guidance  for  the  DoD 
reviews  and  approves  major  defense  acquisition  program  developmental  test 
and  evaluation  plans 

monitors  and  reviews  acquisition  program  developmental  test  and  evaluation 
activities  of  major  defense  acquisition  programs 
reports  to  Congress  annually  on  developmental  test  and  evaluation 
organization,  capabilities,  and  activities 

Cost  Assessment  and  •  serves  as  principal  advisor  to  the  Secretary  of  Defense  and  other  senior 
Program  Evaluation  officials  on  matters  related  to  cost  analysis  and  the  planning  and  programming 

phases  of  the  planning,  programming,  budgeting,  and  execution  system 

•  develops  independent  cost  estimates  for  major  defense  acquisition  programs 
prior  to  major  milestone  decisions  and  updates  independent  cost  estimates 
when  programs  have  exceeded  critical  cost  thresholds,  known  as  Nunn- 
McCurdy  breaches 

reviews  existing  systems  and  methods  for  tracking  and  assessing  operation 
and  support  costs  on  major  defense  acquisition  programs 

develops  analysis  of  alternative  study  guidance  for  major  defense  acquisition 
programs 

approves  the  analysis  of  alternatives  study  plan  for  each  major  defense 
acquisition  program 

assesses  major  acquisition  program  performance  through  independent 
analyses  and  through  the  Defense  Acquisition  Executive  Summary  process 
identifies  the  root  causes  of  cost  growth  and  other  problems  on  programs  that 
experience  a  critical  Nunn-McCurdy  cost  breach 

Note.  This  table  was  created  using  GAO  analysis  of  the  Weapon  Systems  Acquisition  Reform  Act  of 
2009. 


Performance 
Assessments  and  Root 
Cause  Analyses 


Developmental  Test 
and  Evaluation 


In  addition  to  the  new  organizational  requirements,  the  Reform  Act  requires  the  DoD 
to  ensure  that  the  acquisition  strategy  for  major  defense  acquisition  programs  includes 
measures  to  ensure  competition  or  the  option  of  competition  throughout  the  program  life 
cycle.  This  could  include  strategies  such  as  maintaining  two  sources  for  a  system  (dual¬ 
sourcing)  and  breaking  requirements  for  supplies  or  services  previously  provided  or 
performed  under  a  single  contract  into  separate  smaller  contracts  (unbundling  of  contracts; 
Weapon  Systems  Acquisition  Reform  Act  of  2009,  §  202).  Major  defense  acquisition 
programs  are  also  required  to  provide  for  competitive  prototyping — where  two  or  more 
competing  teams  produce  prototypes  before  a  design  is  selected  for  further  development — 
prior  to  Milestone  B  unless  a  waiver  is  properly  granted  by  the  milestone  decision  authority 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  321  - 


(Weapon  Systems  Acquisition  Reform  Act  of  2009,  §  202(a)),2  and  to  meet  the  following 
Milestone  B  certification  requirements,  including:3 

•  Appropriate  trade-offs  among  cost,  schedule,  and  performance  objectives 
have  been  made  to  ensure  the  program  is  affordable; 

•  A  preliminary  design  review  and  formal  post-preliminary  design  review 
assessment  have  been  conducted  and  on  the  basis  of  such  assessment  the 
program  demonstrates  a  high  likelihood  of  accomplishing  its  intended 
mission; 

•  Technology  has  been  demonstrated  in  a  relevant  environment  on  the  basis  of 
an  independent  review  and  assessment  by  the  Assistant  Secretary  of 
Defense  for  Research  and  Engineering; 

•  Reasonable  cost  and  schedule  estimates  have  been  developed  to  execute, 
with  concurrence  of  the  Director  of  CAPE,  the  program’s  product 
development  and  production  plan; 

•  Funding  is  available  to  execute  the  program’s  product  development  and 
production  plan; 

•  The  DoD  has  completed  an  analysis  of  alternatives  with  respect  to  the 
program;  and 

•  The  Joint  Requirements  Oversight  Council4  has  approved  program 
requirements,  including  an  analysis  of  the  operational  requirements. 

The  Reform  Act  also  requires  the  Joint  Requirements  Oversight  Council  to  ensure 
trade-offs  among  cost,  schedule,  and  performance  objectives  are  considered  for  joint 
military  requirements  (Weapon  Systems  Acquisition  Reform  Act  of  2009,  §  201).  The  GAO 
previously  reported  that  the  Council  considered  trade-offs  made  by  the  military  services 
before  validating  requirements,  but  the  military  services  did  not  consistently  provide  high- 
quality  resource  estimates  to  the  Council  for  proposed  programs  in  fiscal  year  2010.  We  also 
found  that  the  Council  did  not  prioritize  requirements,  consider  redundancies  across 
proposed  programs,  or  prioritize  and  analyze  capability  gaps  in  a  consistent  manner  (GAO, 
2011a). 


2  Specifically,  the  Reform  Act  required  the  DoD  to  modify  its  guidance  relating  to  the  operation  of  its 
acquisition  system  to  incorporate  these  competitive  prototyping  provisions.  The  DoD  did  so  through 
Directive-Type  Memorandum  (DTM)  09-027,  Implementation  of  Weapon  System  Acquisition  Reform 
Act  of  2009  (Dec.  4,  2009,  incorporating  Change  3,  Dec.  9,  2011).  Major  defense  acquisition 
programs  are  those  estimated  by  the  DoD  to  require  an  eventual  total  expenditure  for  research, 
development,  test,  and  evaluation,  including  all  planned  increments,  of  more  than  $365  million,  or  for 
procurement,  including  all  planned  increments,  of  more  than  $2.19  billion  in  fiscal  year  2000  constant 
dollars  or  designated  by  the  USD(AT&L).  The  Milestone  Decision  Authority  for  major  defense 
acquisition  programs  is  the  USD(AT&L),  the  head  of  a  DoD  component,  or  if  delegated  the 
component  acquisition  executive. 

3  Pub.  L.  No.  1 1 1-23;  various  sections,  codified  at  10  U.S.C.  §  2366b.  The  Reform  Act  revised  the 
Milestone  B  certification  requirements  for  trade-offs,  preliminary  design,  technology  demonstration, 
and  reasonable  cost  and  schedule  estimates.  The  remaining  Milestone  B  certification  requirements 
bulleted  here  were  unrevised  by  the  Reform  Act. 

4  The  Joint  Requirements  Oversight  Council  is  an  advisory  council  to  the  Chairman  of  the  Joint  Chiefs 
of  Staff  with  the  responsibility  to:  (1)  identify,  assess,  and  approve  joint  military  requirements;  (2) 
assist  acquisition  officials  in  identifying  alternatives  to  acquisition  programs;  and  (3)  assist  the 
Chairman  of  the  Joint  Chiefs  of  Staff  in  assigning  priority  for  joint  military  requirements. 
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Findings 

The  GAO’s  analysis  of  11  weapon  acquisition  programs  showed  the  Reform  Act  has 
reinforced  early  attention  to  requirements,  cost  and  schedule  estimates,  testing,  and 
reliability.  For  example,  prior  to  starting  development,  an  independent  review  team  raised 
concerns  about  the  Ground  Combat  Vehicle  program’s  many  requirements  and  the  risks 
associated  with  its  seven-year  schedule.  Subsequently,  the  Army  reduced  the  number  of 
requirements  by  about  25%  and  prioritized  them,  giving  contractors  more  flexibility  in 
designing  solutions.  In  addition,  the  developmental  test  and  evaluation  office — resulting  from 
the  Reform  Act — used  test  results  to  help  the  Joint  Light  Tactical  Vehicle  program  develop  a 
more  realistic  reliability  goal  and  a  better  approach  to  reach  it.  Shown  in  Table  2  are  areas 
where  the  Reform  Act  influenced  several  programs  in  the  GAO’s  review. 


Table  2.  Reform  Act  Influence  on  Case  Study  Programs 


Program 

Requirements 

Cost  and  schedule 

Testing 

Reliability 

Before  Milestone  B 

Ground  Combat  Vehicle 

V 

V 

V 

✓ 

Joint  Light  Tactical  Vehicle3 

s 

V 

✓ 

Ohio  Class  Replacement 

✓ 

s 

✓ 

Ship  to  Shore  Connector3 

✓ 

✓ 

After  Milestone  B 

Joint  Strike  Fighter 

V 

Global  Hawk 

V 

✓ 

Gray  Eagle 

✓ 

V 

V 

✓ 

KC-46  Tanker 

s 

✓ 

Littoral  Combat  Ship  Seaframe 

s 

Remote  Minehunting  System 

✓ 

V 

✓ 

Small  Diameter  Bomb  II 

✓ 

s 

✓ 

Notes.  This  table  was  created  using  GAO  analysis  of  DoD  data. 


a  During  the  course  of  our  review,  the  Joint  Light  Tactical  Vehicle  and  Ship  to  Shore  Connector 
programs  held  a  Milestone  B  review. 

While  the  DoD  has  taken  steps  to  implement  most  of  the  fundamental  Reform  Act 
provisions,  some  key  efforts  to  date  have  been  primarily  focused  on  the  DoD’s  largest 
weapon  acquisition  programs.  The  DoD  faces  five  challenges — organizational  capability 
constraints,  the  need  for  additional  guidance  on  cost  estimating  and  Reform  Act 
implementation,  the  uncertainty  about  the  sufficiency  of  systems  engineering  and 
developmental  testing  resources,  limited  dissemination  of  lessons  learned,  and  cultural 
barriers  between  the  Office  of  the  Secretary  of  Defense  (OSD)  and  the  military  services — 
that  limit  its  ability  to  broaden  the  Reform  Act’s  influence  to  more  programs.  Service  officials 
believe  additional  guidance  is  needed  to  improve  their  cost  estimates  and  other 
implementation  efforts.  They  also  believe  that  lessons  learned  from  programs  that 
experience  significant  cost  and  schedule  increases  should  be  shared  more  broadly  within 
the  acquisition  community.  These  challenges  seem  straightforward  to  address,  but  they  may 
require  more  resources,  which  have  been  difficult  to  obtain.  Ensuring  the  services  have  key 
leaders  and  staff  dedicated  to  systems  engineering  and  developmental  testing  activities, 
such  as  chief  engineers  at  the  service  level  and  technical  leads  on  programs,  as  well  as 
breaking  down  cultural  barriers  are  more  difficult  to  address.  They  will  require  continued 
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monitoring  and  attention  by  the  USD(AT&L),  service  acquisition  executives,  and  offices 
established  as  a  result  of  the  Reform  Act  to  address. 

Recommendations 

The  GAO  recommends  the  DoD  develop  additional  cost  estimating  and  Reform  Act 
implementation  guidance;  make  lessons  learned  available  to  the  acquisition  community;  and 
assess  the  adequacy  of  the  military  services’  systems  engineering  and  developmental 
testing  workforce.  The  DoD  generally  concurred  with  the  recommendations.  The  GAO 
clarified  one  recommendation  to  make  it  clear  that  the  DoD  needs  to  designate  an  office(s) 
within  the  Acquisition,  Technology,  and  Logistics  organization  to  provide  practical  Reform 
Act  implementation  guidance  to  program  offices. 

For  a  more  detailed  discussion  of  our  findings,  as  well  as  our  scope  and 
methodology,  see  www.gao.gov/products/gao-1 3-1 03. 
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